tensorflow + dalex = :)

For the blog: https://medium.com/responsibleml/tensorflow-dalex-929ca520d3d9

introduction to the topic: Explanatory Model Analysis: Explore, Explain, and Examine Predictive Models

In [1]:
import warnings
warnings.filterwarnings('ignore')

read data

In [2]:
import pandas as pd
pd.__version__
Out[2]:
'1.1.3'
In [3]:
data = pd.read_csv("https://raw.githubusercontent.com/pbiecek/xai-happiness/main/happiness.csv", index_col=0)
data.head()
Out[3]:
score gdp_per_capita social_support healthy_life_expectancy freedom_to_make_life_choices generosity perceptions_of_corruption
Afghanistan 3.203 0.350 0.517 0.361 0.000 0.158 0.025
Albania 4.719 0.947 0.848 0.874 0.383 0.178 0.027
Algeria 5.211 1.002 1.160 0.785 0.086 0.073 0.114
Argentina 6.086 1.092 1.432 0.881 0.471 0.066 0.050
Armenia 4.559 0.850 1.055 0.815 0.283 0.095 0.064
In [4]:
X, y = data.drop('score', axis=1), data.score
n, p = X.shape

create a model

In [5]:
import tensorflow as tf
tf.__version__
Out[5]:
'2.3.0'
In [6]:
tf.random.set_seed(11)

normalizer  = tf.keras.layers.experimental.preprocessing.Normalization(input_shape=[p,])
normalizer.adapt(X.to_numpy())

model = tf.keras.Sequential([
    normalizer,
    tf.keras.Input(shape=(p,)),
    tf.keras.layers.Dense(p*2, activation='relu'),
    tf.keras.layers.Dense(p*3, activation='relu'),
    tf.keras.layers.Dense(p*2, activation='relu'),
    tf.keras.layers.Dense(p, activation='relu'),
    tf.keras.layers.Dense(1, activation='relu')
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(0.001),
    loss=tf.keras.losses.mae
)
In [7]:
model.fit(X, y, batch_size=int(n/10), epochs=2000, verbose=False)
Out[7]:
<tensorflow.python.keras.callbacks.History at 0x21d6ff38760>

explain the model

Explainer initialization communicates useful information

In [8]:
import dalex as dx
dx.__version__
Out[8]:
'0.4.1'
In [9]:
explainer = dx.Explainer(model, X, y, label='happiness')

Preparation of a new explainer is initiated

  -> data              : 156 rows 6 cols
  -> target variable   : Parameter 'y' was a pandas.Series. Converted to a numpy.ndarray.
  -> target variable   : 156 values
  -> model_class       : tensorflow.python.keras.engine.sequential.Sequential (default)
  -> label             : happiness
  -> predict function  : <function yhat_tf_regression at 0x0000021D775B7700> will be used (default)
  -> predict function  : Accepts pandas.DataFrame and numpy.ndarray.
  -> predicted values  : min = 2.86, mean = 5.42, max = 7.73
  -> model type        : regression will be used (default)
  -> residual function : difference between y and yhat (default)
  -> residuals         : min = -0.616, mean = -0.0103, max = 0.555
  -> model_info        : package tensorflow

A new explainer has been created!

model level explanations

firstly, assess the performance

In [10]:
explainer.model_performance()
Out[10]:
mse rmse r2 mae mad
happiness 0.017569 0.132549 0.985729 0.072329 0.03636

which features are the most important?

In [11]:
explainer.model_parts().plot()
In [12]:
explainer.model_parts(type='shap_wrapper').plot()

WARNING:tensorflow:From c:\users\hbani\appdata\local\programs\python\python38\lib\site-packages\shap\explainers\_gradient.py:181: set_learning_phase (from tensorflow.python.keras.backend) is deprecated and will be removed after 2020-10-11.
Instructions for updating:
Simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.

what are the continuous relationships between variables and predictions?

In [13]:
explainer.model_profile().plot(variables=['social_support', 'healthy_life_expectancy',
                                          'gdp_per_capita', 'freedom_to_make_life_choices'])

Calculating ceteris paribus: 100%|███████████████████████████████████████████████████████| 6/6 [00:01<00:00,  3.71it/s]

what about residuals?

In [14]:
explainer.model_diagnostics().plot(variable='social_support', marker_size=5, line_width=3)

predict level explanations

investigate the specific country

In [15]:
explainer.predict_parts(X.loc['Poland'], type='shap').plot()

or several countries

In [16]:
pp_list = []
for country in ['Afghanistan', 'Belgium', 'China', 'Denmark', 'Ethiopia']:
    pp = explainer.predict_parts(X.loc[country], type='break_down')
    pp.result.label = country
    pp_list += [pp]
pp_list[0].plot(pp_list[1::], min_max=[2.5, 8.5])

surrogate approximation

In [17]:
lime_explanation = explainer.predict_surrogate(X.loc['United States'], mode='regression')
lime_explanation.show_in_notebook()

WARNING:tensorflow:From c:\users\hbani\appdata\local\programs\python\python38\lib\site-packages\lime\lime_tabular.py:355: Sequential.predict_proba (from tensorflow.python.keras.engine.sequential) is deprecated and will be removed after 2021-01-01.
Instructions for updating:
Please use `model.predict()` instead.
WARNING:tensorflow:Network returning invalid probability values. The last layer might not normalize predictions into probabilities (like softmax or sigmoid would).

interpretable surrogate model

In [18]:
surrogate_model = explainer.model_surrogate(max_vars=4, max_depth=3)
surrogate_model.performance
Out[18]:
mse rmse r2 mae mad
DecisionTreeRegressor 0.203036 0.450595 0.829172 0.364118 0.322454
In [19]:
surrogate_model.plot()

Plots

This package uses plotly to render the plots:

Resources